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METHODS AND CIRCUITS FOR INTRINSIC PROCESSING 
OF IMAGE DATA WITHIN IMAGE SENSING DEVICES 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The invention relates generally to the field of image processing. More 
specifically, the invention relates to image or motion video compression. 

2. Description of the Related Art 

In the current state of the art, image capture devices, those devices which 
represent an environment or scene by electrical signals that are proportional to the color 
arid/or intensity of light present in the scene, are often manufactured and designed using 
CCD (Charge Coupled Device) technology. A CCD image capture device utilizes small 
photocells to generate electrical signals that are related to the incident light from the 
scene that strikes the photocells. The imaging device contains a two-dimensional array 
of such photocells such that a series of signals across an entire focused upon scene can be 
captured and stored. More recently, CMOS (Complementary Metal Oxide 
Semiconductor) imager devices have been developed which function to provide the same 
sort of output signals that CCD devices to but often at a lower cost and complexity. 
Examples of common imaging devices include scanners, motion video cameras and 
digital still cameras. 

Many of these imaging devices, whether based on CMOS or CCD technology, are 
designed to be compliant with a transmission standard known as NTSC (National 
Television Systems Committee). One stricture of NTSC is that an image should be 
transmitted not in successive scan rows (of the imager array), but with odd rows 
transmitted separately from the even rows. This process of separating odd and even rows 
is commonly referred to as an interlaced scan. An NTSC signal has light intensity 
information encoded as analog voltage levels, color information encoded in the phase 
and amplitude of a color carrier frequency and so on. When an NTSC signal is 
forwarded for image processing to a computer system, the computer system utilizes a 
signal converter to transform the analog encoded information into luminance and 
chrominance digital values for the transmitted image. The most commonly used 
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luminance-chrominance format for the digital representation of images is known as 
YCrCb (a digital color system referred to as the CCER (International Consultative 
Committee on Broadcasting) 601 color space). The conversion from NTSC to YCrCb is 
serial in nature and due to the serial processing character of most commercially available 
microprocessors. Subsequent image processing is also performed in serial. One notable 
exception to the predominance of serial data processing is Intel's MMXfTM) technology 
based processors which use SIMD (Single Instruction Multiple Data) processing. To 
complement the use of such processors in conjunction with imaging devices, it would be 
useful to have parallel processing of the cells used to capture digital values. Further, a 
key factor in the practical application of the digital photocell is that the relatively long 
integration times of the analog photocell portion allows the use of a relatively slow, but 
therefore simple method of digitization. For motion video, which involves certain 
inherently serial operations such per-pixel difference calculations (where the difference 
between pixels and/or frames rather than the original values are encoded), it is useful to 
implement an architecture that allows such calculations to be performed on the imaging 
device rather than strictly through a host processor. 

* 

SUMMARY OF THE INVENTION 
What is disclosed is an apparatus comprising an analog photocell adapted to 
capture light energy incident upon it as an analog signal, a sample-and-hold amplifier 
coupled to the photocell and adapted to store the analog signal and a digital converter 
coupled to the amplifier, the convener transforming the analog signal into a digital value, 
the value proportional to the amount of the light energy. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The objects, features and advantages of the method and apparatus for the present 
invention will be apparent from the following description in which: 

Figure 1 illustrates a digital photocell utilized in the invention. 
Figure 2 is a simplified block diagram of a conventional serial imager. 
Figure 3 illustrates an architecture for more efficient image differencing. 
Figure 4 illustrates one embodiment of the invention. 
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Figure 5 illustrates a per-pixel analog difference engine according to an 
embodiment of the invention. 

Figure 6 illustrates a per-pixel digital difference engine according to an 
embodiment of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 
One aspect of the invention involves enhancing each photocell that is used in the 
imaging array to capture a scene. Rather than using a purely analog photocell it may be 
advantageous, in devices manufactured from CMOS technology, to utilize a combination 
of analog and digital signaling. An analog photocell can be embedded, according to one 
embodiment of the invention, with conversion circuitry to create a digital photocell. The 
digital photocell will convert the analog signal generated by incident light into a digital 
code, allowing it to image processed in the digital domain. An array of such digital 
photocells can be used, as shown in Figure 6, to implement a digital image processing 
system on the imaging device. 

Figure 1 illustrates a digital photocell utilized in the invention. 
An analog photocell 110 captures the light energy incident upon it from the scene 
being imaged. The analog photocell 1 10 operates according to an integration time, T, 
which varies according to ambient light condition. The integration time is less than the 
interval needed to saturate the photocell. The charge accumulated at the photocell is 
input to a sample and hold amplifier 120. When the photocell discharges its charge, a 
counter 140 is reset and begins counting for the next integration period. The counter 140 
is driven by a voltage control oscillator (VCO) 130. VCO 130 has an input level set by a 
previously acquired charge that has been stored in the sample and hold amplifier 120. 
VCO 130 controls the speed at which the counter 140 increases. The greater the light 
intensity at the analog photocell the faster the counter 140 will be driven by VCO 130. 
During an integration period for a particular charge, counter 140 is counting up, and 
before reset, its value is sent to a register 150. The digital value in register 150, which is 
also the final value of counter 140, reflects the intensity value of the pixel in the previous 
integration period. For a motion imaging system, register 150 contains the pixel of one 
"frame" in the imaging. The photoelectric charge representative of the next frame is in 
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sample and hold circuit 120 while the counter 140 is generating the digital value 
representative of the next frame. The register 150 holds the pixel value until it is output 
as part of the image or for further processing. Each of the digital photocells that 
compose the imager pixel array may be regulated using the same timing and control 
sequence since the photocells act in parallel, outputting an entire frame periodically. 

To ensure that the dynamic range of the counter matches the dynamic range of the 
photocells, the sample and hold amplifier can be equipped to scale the input to VCO 120 
as appropriate. The dynamic range may be mismatched due to differing ambient light 
levels in the scene being captured. The variance in integration period that may result 
from a charge in ambient light of the scene ensures that the captured image has the 
proper contrast. To adjust the dynamic range of the VCO 120 to match the analog 
photocell, a global scaling voltage 160 can be applied to the sample and hold amplifier of 
each digital photocell in the array which uniformly adapts the VCO component in each 
photocell to have a dynamic range consistent with the present ambient light conditions. 
The enhanced digital photocell of Figure 1 may be utilized in a serial imaging device, or 
for use in parallel image processing architectures. 

Figure 2 is a simplified block diagram of a conventional serial imager. 

As noted above, motion video compression such as MPEG, utilizes a differencing 
approach to encode successive frames of image data. After two frames are captured or 
imaged completely, the difference between corresponding pixels is computed and this 
difference is then encoded. This allows highly correlated or redundant image features to 
be encoded using the fewest number of bits. For instance, in a video-conferencing 
session, the background of the captured image may change only slightly or not at all from 
one frame instance to the next. In this case, rather than transmitting the entire 
background portion at a successive frame, just the pixel variance between frames could 
be transmitted. In Figure 2, the serial imager utilized in CCD imaging devices would 
shift out pixel information for an entire frame and then another entire frame before the 
first difference frame could be computed. This conventional method required thus the 
capturing and storing of two (or more) entire image frames to generate a third frame 
representing the difference. The first frame, a "key" frame is captured and digitized, as is 
a successive frame. The digitized frames are then differed to generate a difference frame. 



4 



WO 00/69166 



PCT/US00/11963 



To reduce the delay and computational load in conventional image differencing 
apparatus, an architecture similar to that of Figure 3 may be employed. 

Figure 3 illustrates an architecture for more efficient image differencing. The 
conventional design of imaging devices is to perform differencing of successive captured 
frames after they are captured and digitized. To improve upon this conventional design, 
the computational load can be reduced significantly if an analog differencing is done 
prior to digitizing. The parallel-shift differencing apparatus of Figure 3 utilizes shifting 
to achieve the goal of generating image difference data. 

Consider a set of eight exemplary analog photocells A31, A32, A33, A34, A3J, A36. 
A37, and A38. Photocells capture analog light intensity at fixed locations in the scene. 
These intensity values are represented by an amount of charge that accumulates in the 
photocell during its integration time. The photocells A31, ... A38 do not generate a 
digitized output as does the digital photocell unit of Figure 1. Rather, the stored charge 
is passed at the end of the integration period (which is the same for all photocells in a 
given frame instant), to a corresponding shift cell. For each row of analog photocells, 
there are two rows of shift cells. One row of shift cells stores photocell outputs for a first 
frame ("key" frame) while a second row of shift cells stores the photocell outputs of the 
successive frame. Each row of shift cells outputs photocell data serially. 

The row of shift cells for key frame output are designated S32, S34, S36, S38, S40. 
S42, S44 and S 4 6- The row of shift cells storing output for the frame immediately 
succeeding the key frame are labeled S31, S33, S35, S37, S39, S41, S43 and S45. When the 
imaging architecture is first initialized, all shift cells store a null or zero intensity value. 
When the first image frame is captured, each of the analog photocells A31, ... A38 will 
develop a charge representative of light intensity at a particular location in the scene that 
is incident upon the photocell. This set of signals is transferred to the row of shift cells 
S31, S33, ... Sa6- The row of shift cells for keyframe is at that instant, unfilled. Rather 
than outputs this first frame of data, the architecture waits until the next frame is 
captured. When the next image frame is captured by analog photocells A31, ... A38. the 
result of the previous frame is first transferred from shift cells S31, S33, ... S45 to the row 
of shift cells S32, S34, ... S 4 6. respectively, as indicated in Figure 3. Next, at the end of 
the integration period for the second frame, the signals are transferred from analog 
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photocells A 3 i, ... A 38 to the shift cells S 3l , S33. ••• S 45 - At that instant, both rows of shift 
cells contain image frame information. The row of shift cells S 32 , S34, ••• S46 which 
stores the first frame is shifted out. This represents a key frame output 312. Key frame 
output 312 is simultaneously shifted to an input differential op-amp (operational 
amplifier) 310. 

The result of the current frame stored in shift cells S 3I , S 33 , ... S45. is shifted out to 
the other input of differential op-amp 310. Differential op-amp 310 generates an analog 
signal, delta frame output 314, which is the result of previous frame (key frame) 
subtracted from the current frame. The analog signal delta frame output 314 and key 
frame output 312 may both be digitized prior to storage or processing. Since a serial 
shifting operation will output the analog key frame and current frame outputs only pixel 
by pixel, the entire current frame and key frame must first be shifted to output and to 
differential op-amp 310. After the seriaJ shifting operation is complete and the last of the 
key and current frames are output/processed, then the current frame stored in shift cells 
S 3 |, S 33 , ... S45 is shifted in parallel to the row S 32 , S34, ... S 46 and thus, becomes the next 
key frame. 

The advantages of this design lie primarily in the ability to send to digitization 
only an analog difference frame output, rather than two entire image frames. Depending 
on the further down-the-line processing to be performed, the delta frame output 314 
and/or the key frame output 312 may be digitized. In the conventional design, two entire 
frames of analog photocell information is captured, and shifted out separately, after 
which digitizing and differencing are performed. In the architecture of Figure 3, both 
the key frame 312 and the differential for the next frame (delta frame output 314) are 
shifted to output simultaneously. 

The embodiment of Figure 3 still requires the shifting out of an entire key frame 
and difference frame, albeit simultaneously, before another frame can be captured by the 
analog photocell. A further improvement to this architecture is shown in Figure 4 
according to yet another embodiment of the invention. 

Figure 4 illustrates one embodiment of the invention. 

In the embodiment of Figure 4, the current frame is shifted out immediately and 
is also regenerated and fed back by way of an op-amp 420. 
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In the architecture of Figure 4, a current image frame is captured by an array of 
N+l analog photocells Ao, A| ...An which are then passed in parallel to shift cells Co, 
C|...C N , respectively. Shift cells C 0 , C|...C N shift out the captured frame data in a 
cascade (bucket-brigade) fashion from Cn to C 0 . The current frame data is regenerated 
by an op-amp 420 and fed back to an array of shift cells S N ...S 0 as shown. As the 
regenerated current frame is fed back, the current frame is differenced against a previous 
frame shifted out of the array of shift cells So-Sn- The differencing between the current 
frame and the previous frame is accomplished by an op-amp 410 and produces a pixel - 
by-pixel difference frame output. This serial imaging system has the intended advantage 
•of providing both the current frame and a difference frame without having to wait for an 
entire frame of pixel data to be captured. Ordinarily, two frames, a first frame and a 
second frame must both be captured before a difference frame can be generated. The 
architecture of Figure 4 eliminates such a limitation on serial imaging. Op-amps 410 
and 420, though not described in detail, can be designed by one of skill in the an but 
should have the capabilities of boosting signal integrity in the case of op-amp 420 and 
differencing two signals in the case of op-amp 410. 

Figure 5. illustrates a per-pixel analog difference engine according to an 
embodiment of the invention. 

The embodiment of Figure 5 allows the transmission of multiple difference 
frames based upon a key frame. The imaging apparatus would first capture and transmit 
at the image output the key frame where none of the pixels are differential. This key 
frame, shifted out via shift cells Sji, S52, - S54, is also fed back into an array of analog 
holding registers H 5I , H52. ... H54. Prior to being fed back and output, each pixel is 
passed to a regeneration amplifier R5 which regenerates the charge level of the pixel to 
avoid loss in the delay of transmission. This feedback of the current frame into holding 
registers H51...H54 assures that the frame will be available as the "previous" frame when 
the next frame is captured. With the completed frame thus stored, it is possible to 
calculate a difference frame and transmit this difference at the next frame cycle. 

To achieve this, the output each analog holding register H 5 |, H52. and H54 is 
linked to the input of a differential operational amplifier O51, O52, O53 and Om, 
respectively. The current frame is captured by analog photocells A51, A52. A« and A54 
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whose output is passed to other input of differential operational amplifier O51, O52, O53 
and O54, respectively. After the key frame is transmitted, at the output, each subsequent 
frame may be transmitted as the difference relative to the previous frame or key frame as 
computed by amplifiers O51, 0 J2 , O53 and O54. Any number of subsequent "difference" 
frames may be transmitted to the output until the next key frame is desired. The input to 
each shift cell S51, S 52 , S J3 and S54 is one of either the entire original frame captured by 
analog cells A51, A 52 ,A 53 and A54, respectively, (when a key frame is desired) or the 
difference output of the operational amplifiers Oji, O52, 0 53 and O54, respectively. 

A select signal (not shown) is sent to each of a set of analog multiplexers M51, 
M 52 , Ms 3 and M54 which routes either the appropriate key frame data (frame A51, A52, 
A 53 and A Ml respectively) or difference frame data (O51, 0 52 , 0 53 and O54) as desired by 
the application user to shift cells S51, S52, S 53 and S54, respectively. Additional rows of 
shift cells and similar architecture may be linked together one after another as described. 

Figure 6 illustrates a per-pixel digital difference engine according to an 
embodiment of the invention. 

The embodiment of Figure 6 allows the transmission of multiple difference 
frames based upon a key frame. The imaging apparatus would first capture and transmit 
at the image output the key frame where none of the pixels are differential. The initial 
key frame which is captured by digital photocells (pixels) D61, D62, D63 and D64, is 
output on output bus 600. Simultaneously, the digital pixels D61, D62, D63 and D64 is 
fed to a series of digital holding registers H61, H62, H63 and H64, respectively. This 
feedback of the current frame into holding registers H 6 |...H64 assures that the frame will 
be available as the "previous" frame when the next frame is captured. With the 
completed frame thus stored, it is possible to calculate a difference frame and transmit 
this difference at the next frame cycle. 

To achieve this, the output each digital holding register H 6 i, rfo, rfo and is 
linked to the input of a subtraction unit S 6 i, S 62 , S 6 3 and S w . respectively. The current 
frame is captured by digital photocells D 6 i, D 62 . D 6 3 and D w whose output is passed to 
other input of subtraction unit S 6 i, S 62 , S 6 3 and S w , respectively. After the key frame is 
transmitted, at the output, each subsequent frame may be transmitted as the difference 
relative to the previous frame as computed by subtraction units S6i, S6 2 , S6 3 and S^. Any 
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number of subsequent "difference" frames may be transmitted to the output until the next 
key frame is desired. 

The output bus transmits one of either the key frame pixels (from D6i, E>6i- D6.i 
and D^) or difference frame pixels (from subtraction units S$\, S$2, S6i and S^i) 
depending on what the applicant/user desires. Based upon the desired mode, key or 
difference, a select signal (not shown) is sent to each one of digital multiplexers M6i, 
M62. M63 and M&t which then accordingly routes either key frame pixels (from D6i, D62, 
D63 and D^, respectively) or difference frame pixels (from S6i, S62. S63 and 
respectively) as indicated. Additional digital outputs similar to those provided by M6i, 
M62, M63 and M« may be repeatedly constructed for each pixel desired. 

The exemplary embodiments described herein are provided merely to illustrate 
the principles of the invention and should not be construed as limiting the scope of the 
invention. Rather, the principles of the invention may be applied to a wide range of 
systems to achieve the advantages described herein and to achieve other advantages or to 
satisfy other objectives as well. 
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CLAIMS : 

What is claimed is: 

1 . An apparatus comprising: 

an analog photocell adapted to capture light energy incident upon it as an analog 

signal; 

a sample-and-hold amplifier coupled to said photocell and adapted to store said 
analog signal; 

a digital converter coupled to said amplifier said converter transforming said 
analog signal into a digital value, said value proportional to the amount of said light 
energy. 

2. An apparatus according to claim 1 wherein said digital converter includes: 
a voltage controlled oscillator; 

a counter coupled to said oscillator, said oscillator setting the rate of increase of 
said counter, said rate proportional to said stored analog signal. 

3. An apparatus according to claim 2 further comprising: 

a register coupled to said counter, said register receiving said digital value as an 
output of said counter at the end of a predetermined time period. 

4. An apparatus according to claim 2, wherein said digital converter includes: 

a scaling signal supply, said supply adapting the output of said oscillator in a 
dynamic range consistent with ambient lighting to which said photocell is exposed. 

5. An apparatus according to claim 1 utilized in an imaging device. 

6. A system comprising: 

an array of analog photocells; 
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a first array of shift cells, each of said first array shift cells coupled to one of said 
analog photocells; and 

a second array of shift cells coupled to said first array shift cells such that each 
first array shift cell is coupled to one of said second array shift cells. 

7. A system according to claim 6 further comprising: 

a differential operational amplifier having two input terminals, one input terminal 
coupled to the terminating output of said first array of shift cells, the other input terminal 
coupled to the terminating output of said second array of shift cells, said amplifier 
providing a signal representative of the difference between said first array terminating 
output and said second array terminating output. 

8. A system according to claim 7, wherein a set of such signals, said set as large as 
the size of said first array, represent a delta frame of an image. 

9. A system according to claim 7, wherein said second array terminating output 
represents a key frame of an image when said system is first initiated. 

10. A system comprising: 

a first array of shift cells, the output of each of said first array shift cells coupled 
to the input of the next of said first array shift cells; 
a second array of shift cells; and 

an array of analog photocells, each of said photocells coupled to a corresponding 
one of said second array shift cells. 

11. A system according to claim 1 0 comprising: 

a differential operational amplifier, having two input terminals, one input 
terminal coupled to the terminating output of said first array of shift cells, the other input 
terminal coupled to the terminating output of said second array of shift cells, said 
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amplifier providing a signal representative of the difference between said first array 
terminating output and said second array terminating output. 

12. A system according to claim 1 1, wherein a set of such signals, said set as large as 
the size of said first array, represent a delta frame of an image. 

13. A system according to claim 1 1 further comprising: 

a regeneration amplifier having an input terminal coupled to the terminating 
output of said second array of shift cells, the output of said regeneration amplifier 
coupled to the initiating input of said first array of shift cells, said regeneration amplifier 
enhancing the terminating output of said second array of shift cells. 

14. An apparatus comprising: 

a digital photocells, representing the light intensity of an area of an image as a 
pixel value; 

a holding register coupled to said photocell, said register receiving said value; 
and "" ■- ■ 

a subtraction unit coupled to both said photocell and said holding register, the 
subtraction unit differencing a current pixel value of said photocell with a previous pixel 
value as stored in said holding register. 

15. An apparatus according to claim 14 further comprising: 
an output bus; and 

a multiplexer coupled to said subtraction unit and said digital photocell, said 
multiplexer selectively providing one of the output of said subtraction unit and the value 
in said digital photocell to said output bus. 
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